<html>
<head><meta charset="utf-8"><title>s3 retention · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html">s3 retention</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="228675710"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228675710" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228675710">(Mar 03 2021 at 21:56)</a>:</h4>
<p>While I was typing something else up, I looked at the lifecycle policies for the rust-lang-ci2 bucket, and my reading seems to suggest that we're keeping artifacts around for 168 days in a non-accessible (but still costing us storage) state.</p>
<p>Specifically, it looks like the current setup does this:</p>
<ul>
<li>After 168 days, marks any current object as "expired"</li>
<li>After 168 days of being expired, permanently deletes any expired object</li>
</ul>
<p>I think we don't actually need that second period and presumably this would shrink storage 2x, roughly? I'm a bit confused how we ended up here, so maybe I'm misreading? cc <span class="user-mention" data-user-id="121055">@Pietro Albini</span></p>



<a name="228675732"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228675732" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228675732">(Mar 03 2021 at 21:56)</a>:</h4>
<p>This may also impact <span class="user-mention" data-user-id="117568">@Aidan Hobson Sayers</span>'s deletion work I guess</p>



<a name="228676100"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228676100" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228676100">(Mar 03 2021 at 21:59)</a>:</h4>
<p>whelp</p>



<a name="228676726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228676726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228676726">(Mar 03 2021 at 22:03)</a>:</h4>
<p><a href="/user_uploads/4715/mkpGhVJzSUBbLWvGt9EFZUWd/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/mkpGhVJzSUBbLWvGt9EFZUWd/image.png" title="image.png"><img src="/user_uploads/4715/mkpGhVJzSUBbLWvGt9EFZUWd/image.png"></a></div>



<a name="228676893"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228676893" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228676893">(Mar 03 2021 at 22:04)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> yeah, we're storing that data double</p>



<a name="228676939"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228676939" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228676939">(Mar 03 2021 at 22:05)</a>:</h4>
<p>I did some digging and it seems like it was a change made pre-terraform</p>



<a name="228677023"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228677023" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228677023">(Mar 03 2021 at 22:05)</a>:</h4>
<p>this is what I imported in dec 2019: <a href="https://github.com/rust-lang/simpleinfra/commit/53d7f781c9fc76ae26eadcbaa7de65bf65b1fa1c#diff-f9eb6c738fb941b0a03b3bb5f0de47ddc8c7a1a2e69937fb1ea0c4e321387579R13-R19">https://github.com/rust-lang/simpleinfra/commit/53d7f781c9fc76ae26eadcbaa7de65bf65b1fa1c#diff-f9eb6c738fb941b0a03b3bb5f0de47ddc8c7a1a2e69937fb1ea0c4e321387579R13-R19</a></p>



<a name="228677147"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228677147" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228677147">(Mar 03 2021 at 22:06)</a>:</h4>
<p>i'm happy to just... edit the 168 for expired, to, say 7 days or something - or even zero - I think a small buffer can make sense, just in case, but not really more than that</p>



<a name="228677209"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228677209" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228677209">(Mar 03 2021 at 22:07)</a>:</h4>
<p>and the metrics are not showing a doubling in storage usage after that, so it seems like it was configured that way since forever <span aria-label="face palm" class="emoji emoji-1f926" role="img" title="face palm">:face_palm:</span> </p>
<p><a href="https://us-west-1.console.aws.amazon.com/cloudwatch/home?region=us-west-1#metricsV2:graph=~(view~'timeSeries~stacked~false~start~'-P1Y3M~end~'P0D~region~'us-west-1~metrics~(~(~'AWS*2fS3~'BucketSizeBytes~'StorageType~'StandardStorage~'BucketName~'rust-lang-ci2))~period~86400~stat~'Average~yAxis~(left~(min~0)));query=~'*7bAWS*2fS3*2cStorageType*2cBucketName*7d">https://us-west-1.console.aws.amazon.com/cloudwatch/home?region=us-west-1#metricsV2:graph=~(view~'timeSeries~stacked~false~start~'-P1Y3M~end~'P0D~region~'us-west-1~metrics~(~(~'AWS*2fS3~'BucketSizeBytes~'StorageType~'StandardStorage~'BucketName~'rust-lang-ci2))~period~86400~stat~'Average~yAxis~(left~(min~0)));query=~'*7bAWS*2fS3*2cStorageType*2cBucketName*7d</a></p>



<a name="228677270"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228677270" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228677270">(Mar 03 2021 at 22:07)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/s3.20retention/near/228677147">said</a>:</p>
<blockquote>
<p>i'm happy to just... edit the 168 for expired, to, say 7 days or something - or even zero - I think a small buffer can make sense, just in case, but not really more than that</p>
</blockquote>
<p>I don't think you can set it to 0, but 1 should be fine</p>



<a name="228677351"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228677351" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228677351">(Mar 03 2021 at 22:08)</a>:</h4>
<p>ok - any objections to doing that now? I guess the worst case is we delete a bunch of ci artifacts, but seems like it just hurts crater etc</p>



<a name="228677462"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228677462" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228677462">(Mar 03 2021 at 22:09)</a>:</h4>
<p>go for it</p>



<a name="228677645"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228677645" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228677645">(Mar 03 2021 at 22:10)</a>:</h4>
<p>and this also explains why aidan's script removed less than half of the files in the bucket :/</p>



<a name="228677664"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228677664" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228677664">(Mar 03 2021 at 22:10)</a>:</h4>
<p>thanks for catching this mark <span aria-label="heart" class="emoji emoji-2764" role="img" title="heart">:heart:</span></p>



<a name="228678052"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228678052" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228678052">(Mar 03 2021 at 22:13)</a>:</h4>
<p>alright done</p>



<a name="228678098"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228678098" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228678098">(Mar 03 2021 at 22:13)</a>:</h4>
<p>and pushed up a simpleinfra commit</p>



<a name="228678844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228678844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228678844">(Mar 03 2021 at 22:17)</a>:</h4>
<p>I guess we'll see tomorrow roughly, probably the rules run every 24 hours or something</p>



<a name="228680366"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228680366" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228680366">(Mar 03 2021 at 22:29)</a>:</h4>
<p>iirc they stop billing after the rules evaluate every 24 hours, but the actual deletion might happen later</p>



<a name="228680440"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228680440" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228680440">(Mar 03 2021 at 22:29)</a>:</h4>
<p>also filed <a href="https://github.com/rust-lang/simpleinfra/pull/64">https://github.com/rust-lang/simpleinfra/pull/64</a> as a separate cleanup</p>



<a name="228865764"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/228865764" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#228865764">(Mar 04 2021 at 22:59)</a>:</h4>
<p>looks like this saved 30 TB and deleted half the bucket, as expected, at 2 million files</p>



<a name="230384850"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/s3%20retention/near/230384850" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/s3.20retention.html#230384850">(Mar 15 2021 at 17:15)</a>:</h4>
<p>ah yes! I spotted this and made a note and was meaning to raise it but forgot - thanks for following up!</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>